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1.  INTRODUCTION  - This  technical  note  compares  receiver 

operating  characteristic  (ROC)  curves  generated  by  observers 
with  clutter  rate  curves  obtained  by  the  digital  computer  sonar 
simulator  for  6-ping  histories  of  simulated  sonar  returns.  The 
observer's  ROC  curves  were  obtained  by  requiring  the  observers 
to  indicate  their  confidence  in  decisions  that  a signal  was  pre- 
sent on  each  of  several  hundred  film-strip  frames.  Each  frame 

presented  an  independent  simulated  6-echo  cycle  history  at  one 
of  four  s ignal- to-noise  ratios  (S/N) . It  is  shown  that,  for  the 

6-ping  history  situation  at  least,  the  computer  generated  curves 
are  better  than  the  observer's  curves  (in  the  sense  of  showing  a 
larger  probability  of  detection  at  the  same  false  alarm  rate)  at 
low  probabilities  of  detection.  The  situation  is  reversed  for 
high  probabilities  of  detection,  however.  An  explanation  for  this 
behavior  is  given.  ^ 

The  method  of  preparation  of  the  film-strips  has  been 
described^.  The  results  of  the  experimental  observations  are  pre- 
sented in  paragraph  2.  The  manner  of  obtaining  ROC  curves  by 
the  computer  for  the  single-ping  condition  has  been  given  in  detail 
elsewhere  ; a brief  description  is  presented  in  paragraph  3,  how- 
ever. In  paragraph  4 comparison  is  made  of  the  two  types  of  ROC 
curves . 
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2.  THE  OBSERVER'S  ROC  CURVES  - The  observer's  ROC  curves  were 

obtained  by  requiring  the  observers  to  rate  their  responses  of 

"signal  present"  by  their  confidence  that  a signal  was  present. 

Thus,  a rating  of  1 denoted  high  confidence  or  near  certainty  that 

a signal  was  present;  2,  less  assuredness  in  signal  presence;  3, 

low  confidence  in  signal  presence;  4,  a guess  that  a signal  is 

present;  5,  no  signal  is  present.  ROC  curves  were  constructed  by 

plotting  the  cumulative  percent  "hits"  for  a particular  rating 

against  the  corresponding  cumulative  percent  false  alarms.  For 

a particular  rating  the  cumulative  percent  "hits"  is  the  sum  of 

the  percent  hits  for  ratings  of  greater  confidence.  Thus,  for 

example,  for  rating  3 the  cumulative  percent  "hits"  is  Z[P(D)^  + 

P(D)0,  + P(D)o3>  where  P(D) . , i - 1,2,3  is  the  percent  "hits" 

* th  1 

for  the  i racing.  The  percent  false  alarms  are  treated  in  a 
similar  fashion.  A typical  response  sheet  summarizing  the  per- 
formance of  four  observers  for  one  of  the  test  film-strips  is 
shown  in  TABLE  I.  The  values  to  be  plotted  are  underlined. 


The  observers  were  instructed  to  keep  the  number  of  missed 

signals  to  a minimum.  The  observation  time  for  each  frame  of 

the  film-strip  was  6 sec.  This  observation  time  was  used  be- 

3 

cause  it  is  long  enough  not  to  degrade  operator  performance 
yet  short  enough  to  avoid  boredom.  A 3 sec  interval  following 
each  observation  was  allowed  for  the  operators  to  record  their 
responses.  Each  frame  of  the  film-strips  used  in  this  study 
consisted  of  an  independent  six-ping  history  for  a single  sonar 
beam.  About  half  of  the  frames  in  each  film-strip  contained  a 
signal.  The  remaining  frames  contained  noise  alone.  A descrip- 
tion of  the  method  of  preparing  the  test  film-strips  has  been 
given  elsewhere^"  . 
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The  four  observers  used  for  this  study  were  young  adult, 
male  college  students  with  a variety  of  backgrounds.  Some  of 
them  were  the  observers  for  the  work  reported  in  the  technical 
note  cited  in  footnote  3.  The  others  were  given  at  least  five, 
three-hour  training  sessions. 

Each  of  the  film-strips  was  presented  to  the  observers  on 
more  than  one  occasion,  both  to  increase  the  number  of  operator 
decisions  and  to  provide  a measure  of  repeatability  or  consis- 
tency. The  data  presented  in  Table  II  were  obtained  by  totaling 
all  of  the  data  for  each  condition.  The  variation  in  the  data 
(for  a particular  condition),  from  one  viewing  session  to  another, 
was  surprisingly  small.  The  largest  variations  naturally  occurred 
for  the  ratings  denoting  little  confidence  in  a decision  that  a 
signal  is  present.  The  observers  were  scored  on  a YES -NO  basis. 

In  Fig.  1,  RDC  curves  are  presented  for  four  values  of  S/N 
at  the  input  to  the  matched  display.  Three  scales  are  shown 
on  the  abscissa:  the  probability  of  false  alarm,  the  false 
alarm  rate,  and  the  average  time  between  false  alarms.  The 
operators’  , or  observers’ , false  alarm  rate  scale  was  deter- 
mined by  dividing  the  probability  of  false  alarm  by  the  time 
interval  represented  by  the  displayed  information.  Since  each 
of  our  simulated  ping  histories  corresponds  to  some  18  sec  of 
actual  time  and  six  ping  histories  are  shown  simultaneously, 

108  sec  of  data  are  portrayed  in  each  frame. 

It  is  of  interest  to  note  the  marked  change  in  the  ROC  curves 
as  the  S/N  at  the  input  to  the  display  is  varied.  Observe  that 
at  a probability  of  false  alarm  of  0.2,  increasing  the  S/N  at 
the  input  to  the  display  from  8.5  dB  to  10.0  dB  raises  the  pro- 
bability of  detection  from  0.34  to  0.84.  Also  note  that  the 
curve  for  8.5  dB  is  not  greatly  better  than  the  chance  line  re- 
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suiting  from  mere  guessing.  (Here,  signals  are  assumed  to  be 
present  in  half  of  the  trials.)  This  situation  points  out  the 
narrow  range  in  S/N  between  complete  undetectability  and  near 
certainty  of  detection.  When  digital  techniques  are  used  to  analyze 
records  of  noise,  or  signal -plus -noise,  a predetermined  set  of 
decisions  must  be  followed.  Each  sample  value  of  noise  can  be  com- 
pared with  a set  of  thresholds;  a plot  of  the  number  of  sample 
values  exceeding  a given  threshold  as  a function  of  the  threshold 
value  results.  If  the  number  of  independent  sample  values  available 
per  sec,  as  determined  by  the  bandwidth  of  the  original  noise 
function  is  taken  into  account,  a plot  of  the  rate  at  which  the 
threshold  is  exceeded  is  obtained.  Since  a threshold  excession 
normally  produces  a mark  on  the  display,  tending  to  clutter  it, 
the  threshold  excession  rate  will  be  called  the  "clutter  rate.'1 
This  practice  will  avoid  confusion  between  "computer  false  alarm 
rate"  and  the  normal  usage  of  raise  alarm  rate,  in  which  the  pri- 
mary reference  is  to  the  rate  (average  of  course)  at  which  a sonar 
observer  false  alarms. 

Figure  4 is  a plot  of  computer  clutter  rate  as  a function 
of  threshold  setting.  Here  the  threshold  is  measured  in  units  of 
the  standard  deviation  of  the  probability  distribution  of  the 
noise  at  the  scanned  output  of  the  clipped  correlator.  It  will 
be  recalled  that,  in  order  to  match  the  output  of  the  correlator  to 
the  display  cathode  ray  tube,  only  the  largest  of  each  24  suc- 
cessive output  samples  was  retained  and  displayed.  In  this  fashion 
each  of  the  scanned  output  values  could  be  assigned  a separate, 
resolvable  location  on  the  face  of  the  display- tube. 

The  shape  of  the  RDC  curves  reflects  the  shape  of  the  noise 
and  signal-plus-noise  distributions  on  which  the  observers  based 
their  decisions.  The  lack  of  symmetry  about  the  negative  diagonal 
and  the  failure  of  the  slopes  to  have  a value  of  unity  at  the  nega- 
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tive  diagonal  both  indicate  a skewed  distribution.  Jeffress  has 
discussed  these  points  in  some  detail.  Figure  2 is  a plot  of  the 
probability  density  distribution  function  of  the  noise  samples  at 
the  input  to  the  display.  The  skewness  of  this  curve  is  immediately 
obvious.  Since  the  noise  samples,  originally  Gaussian,  were  band- 
pass-filtered, clipped,  correlated,  scanned,  and  averaged,  the  de- 
parture of  the  distribution  from  Gaussian  is  not  surprising. 

In  Figure  3,  the  ROC  curves  have  been  replotted  as  a func- 
tion of  observers'  false  alarm  rate  on  semi-log  paper  in  order  to 
facilitate  a comparison  with  machine  (computer)  clutter  rates  for 
clipped  correlation  processing.  Curves  of  this  type  will  be  re- 
ferred to  as  "modified  ROC  curves".  We  shall  first,  however, 
describe  how  the  computer  clutter  rate  curves  were  generated. 


^L.A.  Jeffress,  "Stimulus -Oriented  Approach  to  Detection," 
JASA,  36,  pp. 766-774,  April  1964. 
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3.  THE  COMPUTER  CLUTTER  RATE  CURVES  - Curves  in  which  the 

probability  of  detection  is  plotted  as  a function  of  false  alarm 
rate  (clutter  rate)  can  be  generated  by  the  digital  computer  sonar 
simulator. 

Three  curves  are  shown  in  Figure  4,  the  single-ping  clutter 
rate,  the  six-ping  unrestricted  scan  clutter  rate,  and  the  six- 
ping  restricted  scan  clutter  rate.  The  unrestricted  scan  allows 
the  signal  position  to  vary  by  one  resolution  interval  in  each 
successive  echo  cycle.  In  the  restricted  scan  case  the  signal 
position  is  not  allowed  to  vary. 

Figures  5-8  are  plots  which  show,  for  the  four  sets  of 
data  viewed  by  the  observers,  the  measured  probability  that  the 
signal- to -noise  ratio  at  the  input  to  the  display  will  exceed  values 
given  on  the  abscissa.  The  (S/N)^  values  indicated  on  these 
figures  are  the  averages  of  all  measured  peak  signal  power  to  rms 
noise  power  ratios  occurring  in  the  data  presented.  (Note:  average 
power  ratio  is  not  equal  to  average  dB) . These  figures  show  the 
probability  distribution  of  signal-plus-noise  at  the  input  to 
the  display.  Again  curves  are  shown  for  both  the  single-ping  and 
six- ping  situations. 

If,  in  Figure  4,  the  threshold  is  converted  to  dB , repre- 
senting a S/N  ratio  which  must  be  exceeded  if  a mark  is  to  be  made 
on  the  display,  Figures  5-8  may  be  mapped  into  modified  ROC  curves. 

In  these  curves,  see  Figures  9-12,  probability  of  detection  is 
plotted  as  a function  of  computer  clutter  rate. 

The  observer's  modified  ROC  curves  are  also  shown  on  Figures 
9-12  so  that  a ready  comparison  with  the  computer  modified  ROC  curves 
may  be  made.  The  curves  for  observer  and  computer  cross,  generally 
in  the  neighborhood  of  507.  probability  of  detection,  but  have  markedly 
different  slopes. 
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4.  COMPARISON  OF  OBSERVER'S  AND  COMPUTER'S  MODIFIED  ROC  CURVES 

The  different  slopes  of  the  observer's  and  computer's  modified 
ROC  curves  can  perhaps  be  explained  in  the  following  way.  At  low 
probabilities  of  detection  the  computer  curves  are  seen  to  be  above 
and  to  the  left  of  the  observer's  curves,  i.e.,  the  computer  is 
performing  better  than  the  human  observer.  This  is  probably  caused 
by  the  ability  of  the  computer  to  sense  threshold  excessions  so 
small  that  they  produce  only  very  faint  marks  on  the  display  - 
marks  so  faint  that  the  observer's  attention  is  not  directed  to 
that  area  of  the  display. 


At  higher  probabilities  of  detection,  on  the  other  hand, 
it  is  seen  that  the  observer's  performance  excels  that  of  the 
computer.  In  this  region  more  large  threshold  excessions  are 
to  be  expected,  resulting  in  brighter  marks  on  the  display.  In 
scanning  the  display  the  observer  may  readily  focus  his  attention 
on  an  alignment  of  marks  and  then  search  for  fainter  marks  to 
verify  his  decision  that  a signal  is  present.  The  computer,  however, 
only  makes  binary  decisions;  either  a threshold  is  exceeded  or  it 
is  not,  no  account  is  taken  of  the  amount  by  which  it  is  exceeded. 

As  to  why  the  observer's  curves  are  to  the  right  of  the 
computer  restricted  scan  curve,  it  is  doubted  that  the  observers 
can  discriminate  a signal  position  offset  of  one  resulution  in- 
terval from  one  echo  cycle  to  the  next.  Although  we  have  presented 
only  zero  range-rate  signals,  which  results  in  an  alignment  of 
marks  perpendicular  to  the  series  of  marks  representing  a single 
echo  cycle  return,  we  do  not  know  and  cannot  determine  if  the 
observers  look  for,  or  can  indeed  tell  if,  exact  perpendicularity 
exists . 

Consider  the  situation  in  which  the  signal  position  can 
vary  from  one  echo  cycle  to  the  next  by  one  bin  position  (one 
resolvable  interval).  The  measured  resolvable  spot  size,  or  mark, 
on  our  simulated  A-scan  display  is  0.02  cm.  (The  length  of  the 
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display  representing  one  echo  cycle  is  10  cm  instead  of  some  10  in 
as  in  the  AN/SQS-26  A-scan  display.  To  the  same  scale,  the  re- 
solution interval  on  the  A-scan  would  be  0.02  in).  If  the  target 
position  is  displaced  or  offset  by  one  resolution  interval  for 
each  successive  echo-cycle,  the  maximum  offset  for  six  echo  cycles 
is  only  0.1  cm  (see  Figure  13). 
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5.  CONCLUSIONS 

It  is  felt  that  a logical  basis  has  been  promulgated  for 
comparing  experimentally  determined  ROC  curves  and  computer  gen- 
erated plots  of  probability  of  detection  as  a function  of  clutter 
rate.  The  narrow  spread  in  signal -to -noise  ratio  at  the  input 
to  the  display  that  separates  the  regions  of  complete  undetecta- 
bility and  certainty  of  detection  in  multi-echo  cycle,  multi-beam 
displays  points  out  the  need  for  careful  control  of  the  displays. 
The  increased  detection  capabilities  of  the  AN/SQS-26  sonar  system 
provided  by  signal  processing  can  be  negated  if  the  displays 
distort  the  input  data. 

The  outputs  of  the  sonar  system  signal  processor  must 
be  stored  without  degradation  of  S/N  and  retrieved  for  use  in 
the  display  without  alternation  of  location.  A continuing  need 
exists  for  improving  the  techniques  of  using  storage  tubes. 

Means  for  normalizing  the  inputs  to  the  storage-tube  screen  and 
the  inputs  to  the  display  must  be  devised,  so  that  an  input  at 
a certain  S-/N  will  have  the  same  significance  to  the  sonar 
observer  at  any  location  on  the  display. 
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